A Survey of Constrained Combinatorial Testing

Combinatorial Testing (CT) is a potentially powerful testing technique, whereas its failure revealing ability might be dramatically reduced if it fails to handle constraints in an adequate and efficient manner. To ensure the wider applicability of CT in the presence of constrained problem domains, large and diverse efforts have been invested towards the techniques and applications of constrained combinatorial testing. In this paper, we provide a comprehensive survey of representations, influences, and techniques that pertain to constraints in CT, covering 129 papers published between 1987 and 2018. This survey not only categorises the various constraint handling techniques, but also reviews comparatively less well-studied, yet potentially important, constraint identification and maintenance techniques. Since real-world programs are usually constrained, this survey can be of interest to researchers and practitioners who are looking to use and study constrained combinatorial testing techniques

Test suite prioritization by switching cost

Test suite generation and prioritization are two main research fields to improve testing efficiency. Combinatorial testing has been proven as an effective method to generate test suite for highly configurable software systems, while test suites are often prioritized by interaction coverage to detect faults as early as possible. However, for some cases, there exists reasonable cost of reconfiguring parameter settings when switching test cases in different orders. Surprisingly, only few studies paid attention to it. In this paper, by proposing greedy algorithms and graph-based algorithms, we aim to prioritize a given test suite to minimize its total switching cost. We also compare two different prioritization strategies by a series of experiments, and discuss the advantages of our prioritization strategy and the selection of prioritization techniques. The results show that prioritization by switching cost can improve testing efficiency and our prioritization strategy can produce a small test suite with a reasonably low switching cost. This prioritization can be used widely and help locate fault causing interactions. The results also suggest that when testing highly configurable software systems and no knowledge of fault detection can be used, prioritization by switching cost is a good choice to detect faults earlier

A discrete particle swarm optimization for covering array generation

Software behavior depends on many factors. Combinatorial testing aims to generate small sets of test cases to uncover defects caused by those factors and their interactions. Covering array generation, a discrete optimization problem, is the most popular research area in the field of combinatorial testing. Particle swarm optimization (PSO), an evolutionary search based heuristic technique, has succeeded in generating covering arrays that are competitive in size. However, current PSO methods for covering array generation simply round the particle’s position to an integer to handle the discrete search space. Moreover no guidelines are available to set PSO’s parameters for this problem effectively. In this paper, we extend the set-based PSO, an existing discrete PSO method, to covering array generation. Two auxiliary strategies (particle reinitialization and additional evaluation of gbest) are proposed to improve performance, and thus a novel discrete PSO (DPSO) for covering array generation is developed. Guidelines for parameter settings both for conventional PSO and for DPSO are developed systematically here. Discrete extensions of four existing PSO variants are developed, in order to further investigate the effectiveness of DPSO for covering array generation. Experiments show that conventional PSO can produce better results using the guidelines for parameter settings, and that DPSO can generate smaller covering arrays than conventional PSO and other existing evolutionary algorithms. DPSO is a promising improvement on particle swarm optimization for covering array generation

Search based combinatorial testing

Search techniques can dramatically change our ability to solve a host of problems in applied science and engineering, many search techniques have been developed and applied successfully in many fields, including search based software engineering (SBSE). As a key problem of combinatorial testing, covering array generation has been widely studied and many search techniques have been applied which can be named as search based combinatorial testing (SBCT). SBCT is a branch of search based software testing (SBST) within SBSE. In this paper, to explore the applicability and effectiveness of SBCT, we design six variants from existing search algorithms: Genetic Algorithm, Particle Swarm Optimization and Ant Colony Algorithm by reversing and randomizing their mechanisms. We study their effectiveness in terms of generating a covering array and compare their performance. Experiments show that these search techniques can work well with distinct performance in covering array generation. We believe that these search techniques can be further improved by fine-tuning their configuration and used in broad ranges of area